Operation Supporting Device, Method and Program

ABSTRACT

Surgical operation supporting apparatus and method is disclosed in which based on a plurality of high-definition tomographic images of an operation site produced before surgery, a three-dimensional model of the operation site is generated, and a surface of the operation site is optically measured during the surgical operation, and further, first position information that represents a three-dimensional position of each of points on the surface of the operation site is acquired. Further, an unexposed portion of the operation site is measured with ultrasonic waves during the surgical operation, and the second position information that represents a three-dimensional position of each of points in the unexposed portion of the operation site is acquired. Moreover, based on the first position information and the second position information, displacement and distortion occurring at each of the points in the operation site are estimated using the generated three-dimensional model. And then, in accordance with the estimated displacement and distortion occurring at each of the points in the operation site, the plurality of high-definition tomographic images of the operation site produced before the surgical operation are corrected and the corrected high-definition tomographic images are displayed.

TECHNICAL FIELD

The present invention relates to a surgical operation supportingapparatus, method and program, and particularly to a surgical operationsupporting apparatus and method that supports a surgical operation bycorrecting a plurality of high-definition tomographic images of anoperation site) which images are picked up before surgery, and bydisplaying these images on display means, and also to a surgicaloperation supporting program used to allow a computer to function as thesurgical operation supporting apparatus.

BACKGROUND OF THE ART

Nuclear magnetic resonance-computed tomography (occasionally referred toas MRI (Magnetic Resonance Imaging) or NMR-CT (Nuclear MagneticResonance-Computed Tomography)) is used to obtain tomographic images ofa living body by utilizing a nuclear magnetic resonance phenomenon of anatomic nucleus having spin in the living body within the static magneticfield. The aforementioned tomography has features of avoiding radiationexposure as in X-ray CT imaging, being free of influence from the bones,providing high-definition tomographic images along an arbitrarydirection, and so on, and therefore, it is used in various medicalfields such as surgery and resting (tomographic images obtained by thenuclear magnetic resonance-computed tomography will be hereinafterreferred to as MRI images).

For example, the boundary between a brain tumor to be removed in anoperation of cranial nerve surgery, and a healthy portion is not clearlyrecognized in a visual observation manner. Therefore, in the operationof cranial nerve surgery, MRI images of the head region are taken inadvance, and an actual operation site are compared with the MRI imagesof the head region in a repeated manner. Thus, the surgical operation isproceeded while making a diagnosis of the boundary between the braintumor and the healthy portion. Further, a human's brain includesfunctionally critical areas (eloquent areas, for example, a pyramidalarea, a sensory area, a linguistic area, a visual area, an auditoryarea, and so on). Thus, distribution of these eloquent areas in whatplace and in what manner is inspected in advance and the state in whichvarious eloquent areas are distributed is displayed as a map on an MRIimage of the head region referred to during the surgery (which map isoccasionally referred to as a functional mapping MRI).

In conjunction with the foregoing, the non-patent document 1 disclosesan optical surgery navigation apparatus that is constructed in such amanner that in the operation of cranial nerve surgery, an MRI image ofthe head region taken before surgery, and a space of an operation siteare made to correspond to each other using a common coordinate systemdue to a position detector using infrared light, and the position of aregion for which a surgical operation is currently performed is detectedand shown on the MRI image.

Further, the non-patent document 2 discloses a navigation apparatus thatis constructed in such a manner that ultrasonic tomographic images arepicked up by an ultrasonic probe during surgery, and the position of theultrasonic probe is detected by infrared light, thereby causing an MRIimage of the head region taken before surgery to correspond to theultrasonic tomographic images taken during surgery, and in the samemanner as in the optical surgery navigation apparatus as disclosed inthe non-patent document 1, an operative region for which the surgicaloperation is currently performed is shown on the MRI image,

Further, the patent document 1 discloses the technique that the positionand orientation of a surgical microscope are detected by an opticalposition measurement system, data processing of enlargement ratio, focaldistance and so on is carried out, and image information such as MRIimages of the head region, cerebral images and so on issuperimpose-displayed in such a manner as to be aligned with real-timeimages taken by a surgical microscope during surgery,

Moreover, the patent document 2 discloses the technique that ahigh-definition MRI image taken before surgery (a preoperative image) isreconfigured into a three-dimensional image, and the three-dimensionalimage is distorted based on a deformation condition for estimateddistortions, and is stored as deformation data, and further, an MRIimage is taken during surgery, and a two-dimensional image of an area ofinterest in the preoperative image is reconfigured into athree-dimensional image, and the similarity to the deformation data iscalculated and optimum deformation data is selected, and an image of anobject from a calibration mirror is superimpose-displayed thereon.

-   Non-patent document 1: Medtronic SNT, “StealthStation”, [online],    [searched on Mar. 2, 2004], Internet<URL:    http://www.stealthstation.com/physician/neuro/library/treon.jsp>-   Non-patent document 2: Medtronic SNT, “SonoNav”.[online],[searched    on Mar. 2, 2004], Internet<URL:    http://www.stealthstation.com/physician/neuro/library/sononav.jsp>-   Patent document 1. JP-A No. 2000-333971-   Patent document 2: JP-A No. 2002-102249

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

However, in the operation of cranial nerve surgery, a brain of a patientis distorted during the surgical operation. Therefore, it is difficultfor a surgeon to make a diagnosis of an actual brain condition (forexample, the position or area of a brain tumor) during surgery with ahigh degree of accuracy even if the surgeon refers to an MRI image ofthe head region taken before the surgical operation. Any of theaforementioned non-patent documents 1 and 2, and patent document 1 givesno consideration to distortions of the brain caused by the surgicaloperation, and each disclose the technique that new information is addedto the MRI image taken before surgery or an MRI image is aligned with areal-time image. Thus, these techniques would become helpful to thesurgery, but do not necessarily contribute to improvement of theprecision of the surgery.

The aforementioned problem can be solved by carrying out pickup of anMRI image at fixed intervals during surgery and updating the MRI imageused for reference during surgery at regular intervals, as in thetechnique disclosed in the patent document 2 or the like. However, inorder to realize this solving method, it is necessary to install an MRIimaging device in an operating room, and also necessary to use surgicalequipment and materials made of non-magnetic materials, and so on. Thisresult in an extremely high cost and many restrictions. Further, a freshproblem arises in which the surgical operation needs to be stoppedduring a pickup operation of MRI images. Moreover, the techniquedisclosed in the patent document 2 also has a disadvantage that whendistortions of an operation site during the surgery differ from theestimated deformation condition, the precision of a displayed imagedeteriorates.

The present invention has been achieved in view of the aforementionedcircumstances, and an object of the present invention is to provide asurgical operation supporting apparatus, a surgical operation supportingmethod and a surgical operation supporting program, which each canrealize presentation of images using a simple structure, which imagesrepresent a state of an operation site during surgery with a high degreeof accuracy.

Means for Solving the Problems

According to a first aspect of the present invention, there is provideda surgical operation supporting apparatus comprising: first acquisitionmeans that optically measures a surface of an operation site duringsurgery and that acquires first position information representing athree-dimensional position of each of points on the surface of theoperation site; second acquisition means that measures an unexposedportion of the operation site with ultrasonic waves during surgery andthat acquires second position information representing athree-dimensional position of each of points in the unexposed portion ofthe operation site; correction means that, based on the first positioninformation acquired by said first acquisition means and the secondposition information acquired by said second acquisition means,estimates displacement and distortion at each of the points in theoperation site using a three-dimensional model generated based on aplurality of high-definition tomographic images of the operation site,which images are taken before surgery, and that corrects the pluralityof high-definition tomographic images; and display control means thatallows the high-definition tomographic images corrected by saidcorrection means to be shown on display means.

According to a second aspect of the present invention, there is provideda surgical operation supporting method comprising: a first step in whichbased on a plurality of high-definition tomographic images of anoperation site taken as an image before surgery, a three-dimensionalmodel of the operation site is generated a second step in which asurface of the operation site is optically measured during surgery, soas to acquire first position information that represents athree-dimensional position of each of points oil the surface of theoperation site, and an unexposed portion of the operation site ismeasured with ultrasonic waves during surgery, so as to acquire secondposition information that represents a three-dimensional position ofeach of points of the unexposed portion in the operation site; a thirdstep in which based on the first position information and the secondposition information acquired by said second step, displacement anddistortion at each of the points in the operation site are estimatedusing the three-dimensional model generated by said first step, and inaccordance with the estimated displacement and distortion at each of thepoints in the operation site, the plurality of high-definitiontomographic images of the operation site taken as images before surgeryare corrected; and a fourth step in which the high-definitiontomographic images corrected by said third step are shown on displaymeans.

According to a third aspect of the present invention, there is provideda surgical operation supporting program that causes a computer, to whichdisplay means is connected, to function as: first acquisition means thatoptically measures a surface of an operation site during surgery andthat acquires first position information representing athree-dimensional position of each of points on the surface of theoperation site; second acquisition means that measures an exposedportion of the operation site with ultrasonic waves during the surgeryand that acquires second position information representing athree-dimensional position at each of points in the unexposed portion ofthe operation site; correction means that, based on the first positioninformation acquired by said first acquisition means and the secondposition information acquired by said second acquisition means,estimates displacement and distortion at each of the points in theoperation site using a three-dimensional model generated based on aplurality of high-definition tomographic images obtained before thesurgery, and in accordance with the estimated displacement anddistortion occurring at each of the points in the operation site,corrects the plurality of high-definition tomographic images of theoperation site, which images are produced before the surgery; anddisplay control means that causes the high-definition tomographic imagescorrected by said correction means to be shown on display means,

Effects of the Invention

According to the present invention, a surface of an operation site isoptically measured during surgery and first position informationrepresenting a three-dimensional position of each of points on thesurface of the operation site is acquired; an unexposed portion of theoperation site is measured with ultrasonic waves during surgery andsecond position information representing a three-dimensional position ofeach of points in the unexposed portion of the operation site isacquired; based on the first position information and the secondposition information, displacement and distortion occurring at each ofthe points in the operation site are estimated using a three-dimensionalmodel generated based on a plurality of high-definition tomographicimages of the operation site, which images are taken before surgery, andthe plurality of high-definition tomographic images are corrected; andthe corrected high-definition tomographic images are shown on displaymeans, thereby making it possible to produce an excellent effect thatpresentation of images each representing a state of an operation siteduring surgery with a high degree of accuracy can be realized using asimple structure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram schematically showing the structure of asurgical operation supporting apparatus.

FIG. 2A is a side view of a surgical microscope equipped with athree-dimensional shape measurement device and a video camera.

FIG. 2B is a bottom plan view of a surgical microscope equipped with athree-dimensional shape measurement device and a video camera.

FIG. 3A is a perspective view showing an internal structure of thethree-dimensional shape measurement device,

FIG. 3B is a perspective view showing an internal structure of thethree-dimensional shape measurement device.

FIG. 4A is an image diagram for illustrating pickup of MRI images andgeneration of a three-dimensional brain model from the MRI images.

FIG. 4B is an image diagram for illustrating pickup of MRI images andgeneration of a three-dimensional brain model from the MRI images.

FIG. 4C is an image diagram for illustrating pickup of MRI images andgeneration of a three-dimensional brain model from the MRI images,

FIG. 5 is a flow chart that shows contents of MRI image displayingprocessing executed by a computer of a surgical operation supportingapparatus.

FIG. 6A is an image diagram for illustrating correction of athree-dimensional brain model based on three-dimensional coordinates ofeach of points on the surface of a brain and each of feature pointscorresponding to an unexposed portion of the brain, in which thethree-dimensional brain model generated from MRI images is shown, in asimplified manner, as a two-dimensional one for purpose of easy viewing

FIG. 6B is an image diagram for illustrating correction of athree-dimensional brain model based on three-dimensional coordinates ofeach of points on the surface of a brain and each of feature pointscorresponding to an unexposed portion of the brain, which diagram showsa three-dimensional brain model in which positions of corresponding nodepoints are corrected based on the three-dimensional coordinates of eachof points on the surface of the brain and each of feature pointscorresponding to the unexposed portion of the brain.

FIG. 6C is an image diagram for illustrating correction of athree-dimensional brain model based on three-dimensional coordinates ofeach of points on the surface of a brain and each of feature pointscorresponding to an unexposed portion of the brain, which diagram showsa three-dimensional brain model corrected by estimating and calculatingpositions of the node points whose positions are not corrected, using afinite element method, with a group of node points indicated by aparenthesis mark at the right side being that of node points whosepositions are estimated and calculated using the finite element method.

BEST MODE FOR CARRYING OUT THE INVENTION

One of embodiments of the present invention will be describedhereinafter in detail with reference to the attached drawings, In thefollowing description, a case example is given when the presentinvention is applied to a surgical operation supporting system used toremove a brain tumor formed within a brain of a patient, which is anoperation site, but the present invention is not limited thereto,

FIG. 1 shows a surgical operation supporting apparatus 10 according toan embodiment of the invention, The surgical operation supportingapparatus 10 includes a computer 12 constituted from a personal computer(PC) and the like. The computer 12 includes CPU 12A, ROM 123, RAM 12Cand an input/output port 12D and these components are connected with oneanother via a bus 12E. Further, connected to the input/output port 12Dare a keyboard 14 and a mouse 16 which are used for a user to inputarbitrary information or give various instructions, a display 18comprised of LCD or CRT and allowing displaying of the arbitraryinformation, a hard disk drive (HDD) 20 and a CD-ROM drive 22,respectively. The display is corresponds to display means according tothe present invention.

A three-dimensional model generation pro-ram for generating athree-dimensional brain model, which will be described later, and an MRIimage display program for carrying out MRI image displaying processingwhich will be described later, are previously installed in the HDD 20 ofthe computer 12.

There are several methods for installing (introducing) thethree-dimensional generation program and the MRI image display programinto the computer 12. For example, a CD-ROM in which thethree-dimensional model generation program and the MRI image displayprogram are recorded together with a setup program is set in the CD-ROMdevice 22 of the computer 12, and when the CPU 12A receives aninstruction to execute the setup program, the three-dimensional modelgeneration program and the MRI image display program are sequentiallyread out from the CD-ROM and written in the HDD 20. Due to various typesof setting being carried out if desired, installation of thethree-dimensional model generation program and the MRI image displayprogram is performed.

Further, connected to the input/output port 12D of the computer 12 arean MRI imaging device 24 that allows high-definition tomographic images(MRI images) of a living body to be taken in an arbitrary direction bythe nuclear magnetic resonance-computed tomography, a three-dimensionalshape measurement device 30 and a video camera 32, which are attached toa surgical microscope 26, and an ultrasonic tomographic device 34 allowspickup of ultrasonic tomographic images of a living body, respectively.The MRI imaging device 24 is a image pickup device that carries outpickup of “a plurality of high-definition tomographic images for anoperation site” according to the present invention before surgery, andis located in an MRI image pickup room provided separately from anoperating room. Incidentally, when MRI image displaying processing,which will be described later, is executed by the computer 12, itsuffices that MRI image data taken by the MRI image pickup device 24before surgery can be acquired from the MRI image pickup device 24.Therefore, the computer 12 may not be connected to the MRI image pickupdevice 24, and MRI image data may also be transferred from the MRI imagepickup device 24 to the computer 12 via any one of various recordingmedia such as CD-R, CD-RW, MO, ZIP, DVD-R, DVD-RW and so on,

The surgical microscope 26 includes a microscope portion 38 shown inFIG. 2. The microscope portion 38 is equipped with an objective lens 40directed toward the lower side of the microscope portion 38 (to thelower side in FIG. 2(A)), and an eyepiece lens 42 disposed so as toproject obliquely upward from the side surface of the microscope portion38. The objective lens 40 is specifically a zoom lens (a focal distancevariable lens) comprised of a plurality of lenses. Although notillustrated, optical components such as a prism that guides lightincident on the objective lens 40 toward the eyepiece lens 42 aredisposed between the objective lens 40 and the eyepiece lens 42. Thus,an optical image of an object formed by the objective lens 40 and theeyepiece lens 42 is visually recognized (stereoscopic viewing) by asurgeon that takes a look inside the eyepiece lens 42 with both eyes. Afocusing mechanism is provided between the objective lens 40 and theeyepiece lens 42, and focusing and zooming of an optical image of theobject is made adjustable by the surgeon operating a foot switch or aswitch attached in the vicinity of a lens barrel of the microscopeportion 38S.

The surgical microscope 26 includes a base portion fixed at apredetermined position within the operating room. One end of an arm 44formed by connecting respective ends of plural rods in a rotatablemanner is connected to the base portion in a rotatable manner. Themicroscope portion 38 is connected rotatably to the other end (a leadingend) of the arm 44 (FIG. 2 shows only one end of the arm 44). Anoperating grip portion 46 is attached to the side surface of themicroscope portion 38, and due to the surgeon holding the grip portion46 to move the microscope portion 38, several connecting portions(joints) of the arm 44 are rotated so as to move the microscope portion38 to a desired position or direct to a desired direction, therebymaking it possible for a surgeon to visually recognize a desired visualfield as an optical image.

Further, a measurement/image-pickup unit 48 having the three-dimensionalshape measurement device 30 and the video camera 32 integrated therewithis mounted on the bottom surface of the microscope portion 38. Themeasurement/image-pickup unit 48 has a box-shaped case body, and thevideo camera 32 is attached to the case body of themeasurement/image-pick unit 43 so as to allow imaging of the lower sideof the microscope portion 38, Further, a rectangular opening is formedon the bottom surface of the case body of the measurement/image-pickupunit 48, and the opening is closed by a light transmission cover 50. Thethree-dimensional shape measurement device 30 is attached at a positioncorresponding to the cover 50 (opening) within the case body of themeasurement/image-pickup unit 48.

As shown in FIG. 3, the three-dimensional shape measurement device 30 isequipped with a movable base 56 straddling between a pair of rails 54. Aball screw 60 extending in parallel with the rails 54 and rotated by amotor 58 screws with the movable base 56, so that the movable base 55 ismoved to slide along the rails 54 due to the rotation of the ball screw60. Further, the movable base 56 is provided with a light emittingportion 62 including a laser light source. Disposed sequentially at theside from which laser light (outgoing laser light) is emitted from thelight emitting portion 62 are a mirror 64 mounted on the movable base56, and a galvanometer mirror 66 mounted on a rotating shaft of a motor72 and turned over due to driving of the motor 72. The outgoing laserlight emitted from the light emitting portion 62 is reflected by themirror 64 and the galvanometer mirror 66, so as to be transmittedthrough the cover 50 and emitted out of the case body of themeasurement/image-pickup unit 48,

Further, the outgoing laser light emitted out of the case body of themeasurement/image-pickup unit 48 is reflected by an object to beirradiated (for example, the surface of a brain that is an operationsite), and the reflected light, i.e., return laser light, is transmittedthrough the cover 50 and made incident on a mirror 67. The mirror 67 ismounted on the rotating shaft of the motor 72 along the same directionas that of the galvanometer mirror 66, and is constructed so that thedirection thereof is changed due to the driving of the motor 72.Disposed sequentially at the side from which return laser light isemitted from the mirror 67 are a mirror 68, a lens 69 and a line sensor70 having a large number of photoelectric transfer elements arranged inone row. The return laser light made incident on the mirror 67 isreflected by the mirrors 67 and 68 and made transparent through thelaser 69, and subsequently received by the line sensor 70. An outputsignal from the line sensor 70 is inputted to a controller of thethree-dimensional shape measurement device 30 via an amplifier or an A/Dconverter (which amplifier and A/D converter are both omitted in thedrawings). Further, connected to the controller are a position sensorthat detects the position of the movable base 56, and an angle sensorthat detects the direction of the galvanometer mirror 66 (and the mirror67).

The controller makes a determination as to whether laser light isreceived by which photoelectric transfer element of the line sensor 70based on light-receiving data inputted from the line sensor 70 via theamplifier or A/D converter. And then, based on the position of thephotoelectric transfer element having received laser light on the linesensor 70, the position of the movable base 56 detected by the sensor,and the direction of the galvanometer mirror 66, three-dimensionalcoordinates of a position on the object to be irradiated, at which laserlight is irradiated (specifically, three-dimensional coordinates in athree-dimensional coordinate system set with the position of the casebody of the measurement/image-pickup unit 48 serving as a reference(which coordinate system is referred to as a box type coordinatesystem)) is detected (calculated) using a triangulation method. Further,the motors 72 and 58 are each connected to the controller, and bydriving the motor 72 to change the direction of the galvanometer mirror66 (and the mirror 67), so as to allow the laser light irradiationposition on the object to be irradiated to move along a directionorthogonal to the axis line of the rotating shaft of the motor 72 (mainscan) and further by moving the motor 58 to move the movable base 56,the laser light irradiation position on the object to be irradiated ismoved along a direction parallel to the rails 54 (sub-scan).

As a result, the surface configuration of the object to be irradiated(three-dimensional coordinates of each of spots on the surface of theobject to be irradiated) is entirely measured by the three-dimensionalshape measurement device 30. When tie three-dimensional shapemeasurement device 30 receives an instruction from the computer 12, thethree-dimensional shape measurement device 30 carried out measurement ofthe surface configuration of the object to be irradiated, and outputs,to the computer 12, data representing three-dimensional coordinates ofeach spot on the surface of the object to be irradiated, which data isobtained by the measurement (such data is hereinafter referred to assurface measurement data). The surface measurement data corresponds tofirst position information according to the present invention. The rails54, the movable base 56, the motor 58, the ball screw 60, the lightemitting portion 62, the mirror 64, the galvanometer mirror 66 and themotor 72 collectively correspond to a scanning device as defined inclaim 2, and the mirrors 67, 68, the lens 69, the line sensor 70 and themotor 72 collectively correspond to detecting means as defined in claim2. Further, the position and direction of the video camera 32 areadjusted so as to allow imaging of the same area as that measured by thethree-dimensional shape measurement device 30.

As also shown in FIG. 1, connected to the ultrasonic tomographic device34 is a probe 36 that transmits ultrasonic waves and receives ultrasonicwaves reflected by any object. The ultrasonic tomographic device 34converts a sign 31 inputted from the probe 36 after the probe 36receives the ultrasonic waves, to an ultrasonic tomographic image, andoutputs the same to the computer 12. Further, the probe 36 includesmarks 36A which are made from materials having a high lightreflectivity, attached respectively at the leading end and the rear endthereof, so as to detect the position and direction of the probe 36. Aswill be described later, when ultrasonic tomographic images are taken bythe ultrasonic tomographic device 34, the three-dimensional coordinatesof the mark 36A attached to the probe 36 are measured by thethree-dimensional shape measurement device 30.

Next, the operation of the present embodiment will be described. In thepresent embodiment when a surgical operation for removing a brain tumoris performed, first: MRI images of a head region of a patient (a patienttargeted for the surgical operation) are taken in advance by the MRIimaging device 24 in the MRI imaging room. At the time of taking the MRIimage, as shown in FIG. 4B, based on a skin cut line (a line thatrepresents a position at which the scalp is cut open) which isdetermined in advance for the head region of the patient, three or morepreoperative marks 80 comprised of materials coming out well in an MRIimage are attached at the position in the head region of the patient canthe periphery of an area to be cut open during surgery. The preoperativemarks 80 correspond to first marks as defined in claim 8, and forexample, spherical type white marks of approximately 5 mm in diametercan be used. Incidentally, the area for which cranial surgery isperformed, as shown in FIG. 4B, represents an area in which a portion ofthe skull is removed during surgery.

As shown in FIG. 4A for instance, an MRI image is taken by the MRIimaging device 24 for each of a plurality of cross-sections set atregular intervals (e.g., 1 mm or thereabouts) for the head region of thepatient. As a result, a plurality of MRI images (a plurality oflight-definition tomographic images at the operation site) in which theplurality of cross-sections are made visible with high degree ofdefinition. Some of the plurality of MRI images obtained by the imagingprocess includes the preoperative marks 80 coming out therein. Further,the preoperative marks 80 applied to the head region of the patientremain unchanged from their attachment positions by the time of surgery.Moreover, the plurality of MRI images taken by the MRI imaging device 24correspond to a plurality of high-definition tomographic imagesaccording to the present invention (MRI images as defined in claim 6).

The plurality of MRI images obtained by the aforementioned imaging isinputted from the MRI imaging device 24 to the computer 12, and storedin the HDD 20. Then, a three-dimensional model of a patient's model(three-dimensional brain model) is generated by the computer 12,Specifically, first, MRI images each including at least one of the threepreoperative marks 80 are all selected from among the plurality of MRIimages represented by the inputted data, and a three-dimensionalcoordinate system (hereinafter referred to as an MRI coordinate system)in which the positions of the three preoperative marks 80 on each of theselected MRI images are set as a reference (for example, any one of thethree preoperative marks 80 is determined as an original point).Further, image areas corresponding to the brain of a patient's body areextracted from the plurality of MRI images, a large number of featurepoints which are located on the surface of the brain or at the innerside thereof and which facilitate diagnosis on the MRI image, surfacemeasurement data, or ultrasonic tomographic image (including pointswhich correspond to feature portions of the brain, e.g., brain grooves,gyrus of brain, arteries, veins and so on, and points which correspondto the boundary between a brain tumor and a healthy portion) are set forimage areas extracted from the plurality of MRI images, and thethree-dimensional coordinates of each feature point in the MRIcoordinate system are obtained. Then, the three-dimensional coordinatesof each feature point in the MRI coordinate system, and the positions ofseveral feature points in the MRI image are stored in the HDD 20 or thelike.

Subsequently, feature points (node points) located on the surface of thebrain from among the large number of feature points as set above areconnected by edge lines, and a portion enclosed by the edge lines isregarded as a flat surface. Due to this, a stereoscopic model thatrepresents an outer edge of the brain is generated. Further, the featurepoints (node points) located inside the brain are also connected by edgelines, and a portion enclosed by the edge lines is regarded as a flatsurface. Due to this, a stereoscopic model that represents an outer edgeof the brain is divided into a large number of stereoscopic elements. Asa result, as is also shown in FIG. 4C, a three-dimensional model of apatient's brain in which the patient's brain is represented as a set ofa large number of stereoscopic elements can be generated from theplurality of MRI images of the head region of the patient. Further, inthe computer 12, the degree of density of node points in thethree-dimensional brain model is inspected based on thethree-dimensional coordinates of each feature point (node point) in theMRI coordinate system, and when a region in which spaces between nodepoints are large (i.e., a low density region) exists in thethree-dimensional brain model, node points are added to this region, soas to uniform the size of stereoscopic elements constituting thethree-dimensional brain model. The computer 12 allows data of thegenerated three-dimensional brain model to be stored in the HDD 20.

Incidentally, the aforementioned three-dimensional brain model isgenerated by a computer different from the computer 12, and data of thegenerated three-dimensional brain model may also be transferred to thecomputer 12.

The surgical operation for removing a brain tumor is performed after theaforementioned pickup of MRI images and generation of thethree-dimensional brain model are completed. At the start of thesurgical operation, when a surgeon gives an instruction to the computer12 to activate an MRI image display program, the MRI image displayprocessing is executed by the computer 12 during surgery. The MRI imagedisplay processing will be hereinafter described with reference to theflow chart shown in FIG. 5.

At step 100, it is determined whether or not cranial surgery of apatient has been completed. The determination at step 100 is repeateduntil the decision at step 100 is made affirmative. In the brain tumorremoval operations first, the scalp of a patient is cut open and thebones of skull are exposed. Thereafter, a portion of the exposed bonesof skull, which portion corresponds to a previously-determined area forwhich cranial surgery is to be performed, is removed. Thus, the cranialsurgery for exposing the brain which is an operation site is performed.When information that represents that the cranial surgery has beencompleted is inputted by the surgeon via the keyboard 14, the decisionat step 100 is made affirmative and the process proceeds to step 102, inwhich the surgeon is urged to apply intraoperative marks due to, forexample, a message being shown on the display 18, At step 104, it isdetermined whether or not application of intraoperative marks has beencompleted. The determination at step 104 is repeated until the decisionat step 104 is made affirmative.

When the surgeon is urged to apply intraoperative marks, as is alsoshown in FIG. 4B for instance, the surgeon applies three or moreintraoperative marks 82 on the bones of skull in the vicinity of a bonecavity formed by removing a portion of skull in the cranial surgery. Asthe intraoperative marks 82, spherical type white marks of 5 mm orthereabouts in diameter can be used in the same manner as theaforementioned preoperative marks 80, When application of theintraoperative marks 82 has been completed, the surgeon moves themicroscope portion 38 of the surgical microscope 26 located at aposition where it does not hinder the surgical operation during cranialsurgery, to a position at which an exposed region of the brain comeswithin the visual observation field in which the objective tens 40 andthe eyepiece lens 42 of the surgical microscope 26 form an optical image(accompanied with the movement of the microscope portion 38, the exposedregion of the brain, intraoperative marks 32 and preoperative marks 80come within the measurement range of the three-dimensional shapemeasurement device 30 and also within the image pickup range of thevideo camera 32), and thereafter, the surgeon inputs, via the keyboard14, information representing that application of the intraoperativemarks 82 has been completed.

As a result, the decision at step 104 is made affirmative, and in theprocess subsequent to step 106, calibration processing is carried outwhich obtains a coordinate conversion system used to convert acoordinate value in a box type coordinate system to that in the MRIcoordinate system, That is to say, first, at step 106, a messageindicating that the current state is a “measuring state” is shown on thedisplay 18, whereby the surgical operation is stopped. Further, at step108, an instruction for measurement of a surface configuration is givento the three-dimensional shape measurement device 30, and an instructionfor image pickup of the surface of a brain is given to the video camera32. As a result) in the three-dimensional shape measurement device 30,the outgoing laser light is emitted toward the patient's head includingthe surface of the brain, detection (calculation) of thethree-dimensional coordinates of the position at which laser light isirradiated on the basis of the position at which the return laser lightreflected by the head region of the patient is received by the linesensor 70 is repeated while changing the direction of the galvanometermirror 66 (and the mirror 67) and moving the movable base 56, wherebythe surface configuration of the head region of the patient, for whichcranial surgery has been performed (the three-dimensional coordinates ofeach point in the head region including the surface of the brain) ismeasured. Further, the video camera 32 carries out image pickup of eachpart of the surface of the brain. The aforementioned measurement of thesurface configuration by the three-dimensional shape measurement device30 and image pickup of the video camera 32 are completed in 20 secondsor thereabouts.

At step 110, surface measurement data obtained by carrying outmeasurement of the surface configuration of the head region of apatient, for which cranial surgery has been performed, in thethree-dimensional shape measurement device 30 is fetched in from thethree-dimensional shape measurement device 30, and image data obtainedby carrying out image pickup using the video camera 32 is fetched infrom the video camera 32, At step 112, respective data corresponding tothe preoperative marks 80 and respective data corresponding to theintraoperative marks 82 are extracted from the surface measurement datafetched in from the three-dimensional shape measurement device 30 (thepreoperative marks 80 and the intraoperative marks 82 are detected, bythe three-dimensional shape measurement device 30, as spherical typeobjects), and based on the extracted data, the respectivethree-dimensional coordinates of the center of the preoperative marks 80and the center of the intraoperative marks 82 are obtained bycalculation.

Incidentally, the preoperative marks 80 and the intraoperative marks 82are found as a circular image area in an image obtained by image-pickupof the video camera 32, Therefore, due to the centers of the sphericaltype objects corresponding to the preoperative marks SO andintraoperative marks 82, which marks are represented by data extractedfrom the surface measurement data, and the center of circular imageportions corresponding to the preoperative marks 80 and intraoperativemarks 82, which marks are found in the obtained image, being caused tooverlap with each other, the surface measurement data and the pickupimage can be disposed in a superimposed manner. Further, thethree-dimensional coordinates of each of the preoperative marks 80 areobtained before cranial surgery (before the intraoperative marks 82 areapplied), and the positional relationship of the individual preoperativemarks 80 indicated by the respective three-dimensional coordinates ofthe preoperative marks 80 calculated at step 112 (i.e., the spacesbetween the preoperative marks 80), and the positional relationship ofthe individual preoperative marks 80 obtained before the cranial surgeryare compared with each other, whereby it is confirmed whether or not thepositions of the preoperative marks 80 have been changed accompanied bythe cranial surgery. And then, if necessary, position correction of thepreoperative marks 80 and re-derivation of the respectivethree-dimensional coordinates of the preoperative marks 80 and theintraoperative marks 82 may also be carried out.

The three-dimensional coordinates of each of the preoperative marks 80and intraoperative marks 82 calculated at step 112 are a coordinatevalue in the coordinate system, but the respective three-dimensionalcoordinate values of the preoperative marks 80 in the MRI coordinatesystem are well known. Therefore, at the next step 114, based on thepositional relationship between the group of preoperative marks and thegroup of intraoperative marks, which is represented by the respectivethree-dimensional coordinates of the preoperative marks 80 and theintraoperative marks 82 calculated at step 112, and also based on thecoordinate values of the preoperative marks 80 in the MRI coordinatesystem, a coordinate conversion expression used to convert thethree-dimensional coordinate values in the box type coordinate system tothe three-dimensional coordinate values in the MRI coordinate systemwith the positions of the intraoperative marks 82 serving as areference, and the derived coordinate conversion expression is stored inthe HDD 20 Thus: the calibration processing is completed.

In the present embodiment, the preoperative marks 80 are applied ontothe scalp of the head region of a patient, and therefore, it ispossibility that the positions of the preoperative marks 80 may changewith the proceeding of surgery. However, the intraoperative marks 82 areapplied onto the bones of skull in the vicinity of a bone cavity,thereby preventing the possibility that the positions of theintraoperative marks 82 may change during surgery, In the presentembodiment, the coordinate conversion expression in which thethree-dimensional coordinate values in the box type coordinate systemare converted to the three-dimensional coordinate values in the MRIcoordinate system with the positions of the intraoperative marks 82serving as reference is derived as described above, and therefore, dueto the aforementioned coordinate conversion expression being used, thethree-dimensional coordinate values in the box type coordinate systemcan be converted to the three-dimensional coordinate values in the MRIcoordinate system with the positions of the intraoperative marks 82serving as reference (the MRI coordinate system with the initialpositions of the preoperative marks 80 serving as reference) withoutbeing affected by the state in which the positions of the preoperativemarks 80 changes with the proceeding of surgery, and it is possible tocarry out high-precision positioning of the three-dimensional brainmodel (and the MRI images), the first position information (surfacemeasurement data) and the second position information (unexposed areadata which will be described later in detail).

Further, the three-dimensional coordinate values in the box typecoordinate system can be converted to those in the MRI coordinate systemwith the positions of the intraoperative marks 82 serving as reference,and at the same time, in the subsequent measurement process of thethree-dimensional shape measurement device 30 and in the subsequentimaging process of the video camera 32, the preoperative marks 80applied at positions which are relatively separated from the region forwhich cranial surgery is performed (a bone cavity) do not need to beshown within the measurement range of the three-dimensional shapemeasurement device 30 and within the imaging range of the video camera32. As a result, with the microscope portion 38 (the Three-dimensionalshape measurement device 30 and the video camera 32) being moved closerto the brain, that is, an operation site, the measuring operation of thethree-dimensional shape measurement device 30 and the image pickupoperation of the video camera 32 can be carried out, thereby making itpossible to improve the precision in the measuring operation of thethree-dimensional shape measurement device 30 and in the image pickupoperation of the video camera 32.

At the next step 116, the message indicative of a “measuring state”shown on the display 18 is deleted, and data of an MRI image takenbefore surgery is read from the HDD 20, and based on the read data, theMRI image (a high-definition tomographic image of the brain of apatient) is shown on the display 18. By referring to the aforementionedMRI image shown on the display 18, the surgeon can correctly makediagnosis of the position of a brain tumor to be removed, and so on atthe stage immediately after completion of the cranial surgery.Incidentally, a high-definition display that is exclusively used todisplay MRI images may also be provided for displaying the MRI images.Further, but only an MRI image is shown on the display 18, but a regionon which the surgeon's eyes are kept may also be clearly shown on an MRIimage in such a manner that a calculation as to the center of the visualobservation field in which an optical image is formed by the objectivelens 40 and the eyepiece lens 42 of the surgical microscope 26corresponds to which position on the MRI image, and a blinking mark orthe like is shown at the calculated position on the MRI image.

At the stage immediately after completion of the cranial surgery, theinvention is not limited to displaying of an MRI image (uncorrected MRIimage) taken before surgery. In consideration of the possibility thatthe brain may be displaced and distorted due to the cranial surgery,even at the stage immediately after completion of the cranial surgery,an MRI image corrected through the processes of step 122 to step 150,which will be described later, may be displayed. Alternatively, at thestage immediately after completion of the cranial surgery, it may bepossible for the surgeon to display an uncorrected MRI image or acorrected MRI image.

At the next step 118, it is determined whether or not a timing when theMRI image shown on the display 18 needs to be updated has come. Thisdecision may be given by making a determination as to whether or not afixed period of time has passed after displaying of the MRI image starts(or after previous updating of the MRI image), or by making adetermination as to whether or not the surgeon has given an instructionto update the MRI image. If the decision at step 118 is made negative,the process proceeds to step 120, in which it is determined whether ornot the surgical operation has been completed. This decision can begiven by making a determination as to whether or not information thatindicates completion of surgery has been inputted by the surgeon via thekeyboard 14. If the decision at step 120 is also made negative, theprocess returns to step 118, and steps 118 and 120 are repeated untilany one of these decisions is made affirmative.

If an MRI image is shown on the display 18 in place of the messageindicative of a “measuring state”, as described above, the surgeonstarts the surgical operation subsequent to the cranial surgery in thesurgical removal of a brain tumor. This surgical operation includesoperations for pressing the brain using a scoop, cutting open orremoving a part of the brain, and so on. When these operations areapplied to the brain, various parts of the brain are displaced ordistorted. Therefore, the actual state of the brain (the position orshape of various parts) becomes different from the state of the brainrepresented by the MRI image shown on the display 18. Thus, even if theMRI image shown on the display 19 is referred to, it becomes difficultfor the surgeon to make a precise diagnosis of the position or range ofthe brain tumor to be removed. Accordingly, in the MRI image displayingprocessing, if a fixed period of time has passed after displaying of theMRI image starts (or after previous updating of the MRI images isperformed), or if the information that gives an instruction to updatethe MRI image is inputted by the surgeon via the keyboard 14, thedecision at step 120 is made affirmative and the process proceeds tostep 122. In the process subsequent to step 122, processing forcorrecting and updating the MRI images shown on the display 18 iscarried out.

In other words, first, at step 122, the message indicating that thecurrent state is a “measuring state” is shown on the display 18, and thesurgical operation is thereby stopped. Further, at step 124, aninstruction is given to the three-dimensional shape measurement device30 to measure the surface configuration, and an instruction is given tothe video camera 32 to take an image of the surface of a brain. As aresult, in the three-dimensional shape measurement device 30, theoutgoing laser light is emitted toward a patient's brain including thesurface of the brain, and detection (calculation) of thethree-dimensional coordinates of the position at which laser light isirradiated, based on the position at which the return laser lightreflected by the brain of a patient is received by the line sensor 70,is repeated while changing the direction of the galvanometer mirror 66(and the mirror 67) and moving the movable base 56. Thus, the measuringoperation of the surface configuration of the head region of thepatient, for which the cranial surgery has been performed (thethree-dimensional coordinates of each of points in the head region) iscarried out. Further, the video camera 32 carries out an image pickupoperation for the surface of the brain. The aforementioned measuringoperation of the surface configuration by the three-dimensional shapemeasurement device 30 and the image pickup operation of the video camera32 are completed in approximately 20 seconds.

Incidentally, in the surgical removal of a brain tumor is performed insuch a manner that the surgeon moves the microscope portion 38 byholding the grip portion 46 of the surgical microscope 26, and visuallyrecognizes a region targeted for the surgical operation. Thethree-dimensional shape measurement device 30 and the video camera 32are mounted on the surgical microscope 26, and therefore, when thesurface configuration is measured by the three-dimensional shapemeasurement device 30 or the image pickup operation is carried out bythe video camera 32, it is not necessary to further adjust themeasurement range of the surface configuration or the image pickuprange. The three-dimensional shape measurement device 30 allowsmeasurement of the surface configuration within the measurement rangeincluding the surface of the brain and the intraoperative marks 82 onlyby carrying out measurement of the surface configuration for a fixedrange in the box type coordinate system. Further, the video camera 32also allows imaging of an image pickup range including the surface ofthe brain and the intraoperative marks 82 only by carrying out pickup ofimages for a fixed image pickup range.

At step 126, the surface measurement data obtained by measurement of thethree-dimensional shape measurement device 30 is fetched in from thethree-dimensional shape measurement device 30, and image data obtainedby image pickup of the video camera 32 is fetched in from the videocamera 32. At step 128, data corresponding to the individualintraoperative marks 82 is extracted from the surface measurement datafetched in from the three-dimensional shape measurement device 30, andbased on the extracted data, the three-dimensional coordinates of thecenter of the intraoperative marks 82 are obtained by calculation. Atstep 130, the coordinate conversion expression derived at the previousstep 114 is read out from the HDD 20, and by using the read coordinateconversion expression, the three-dimensional coordinates of each ofpoints on the surface of the brain, which points are represented by thesurface measurement data (the coordinate values in the box typecoordinate system) are converted to the three-dimensional coordinatevalues in the MRI coordinate system, respectively, with the positions ofthe intraoperative marks 82 represented by the three-dimensionalcoordinates obtained at step 128 serving as references, and the surfacemeasurement data after the coordinate conversion is stored in the HDD20. As a result, alignment of the first position information (surfacemeasurement data) and the three-dimensional brain model (and the MRIimage) is completed.

At step 132, due to a message that gives an instruction to the surgeonto produce ultrasonic tomographic images being shown on the display 18,the ultrasonic tomographic images of the brain are produced by theultrasonic tomographic device 34, and an instruction for measurement ofthe surface configuration is given to the three-dimensional shapemeasurement device 30. As a result, the operator holds the probe 36, andgives an instruction to the ultrasonic tomographic device 34 to produceultrasonic tomographic images with the leading end of the probe 36 beingdirected toward the brain of the patient.

When an instruction for pickup of ultrasonic tomographic images isgiven, in the ultrasonic tomographic device 34, the operation oftransmitting ultrasonic waves from the leading end of the probe 36,converting an electric signal outputted from the probe 36 in accordancewith the ultrasonic waves reflected by an arbitrary object and receivedby the probe 36, to digital data, and further causing the data to bestored in memory or the like is carried out in a repeated manner whilechanging the outgoing direction of the ultrasonic waves from the leadingend of the probe 36 along a fixed direction, and thereafter, the datastored in the memory or the like is reordered, thereby allowinggeneration of data that represents an ultrasonic tomographic image ofthe brain for a cross-section parallel to the aforementioned fixeddirection, Further, the surgeon repeatedly gives an instruction to theultrasonic tomographic device 34 to produce ultrasonic tomographicimages, while moving the probe 36 by a substantially fixed distance at atime in a direction substantially orthogonal to the aforementioned fixeddirection.

As a result, a plurality of ultrasonic tomographic images correspondingto plural cross-sections apart from one another substantially at regularintervals are respectively taken for the brain of a patient. Theoperation of taking a plurality of ultrasonic tomographic images iscompleted in approximately three minutes. Further, during the operationof taking ultrasonic tomographic images corresponding to variouscross-sections as described above, due to the surface configurationbeing continuously measured by the three-dimensional shape measurementdevice 30, the position of the probe 36 (three-dimensional coordinatesof the marks 36A attached to the probe 36), and the positions of theintraoperative marks 82 are repeatedly measured,

At step 134, data of a plurality of ultrasonic tomographic images takenby the ultrasonic tomographic device 34 is fetched in from theultrasonic tomographic device 34, and the surface measurement dataobtained by measurement of the three-dimensional shape measurementdevice 30 is fetched in from the three-dimensional shape measurementdevice 30. At step 136, data corresponding to each of the marks 36A ofthe probe 36 at the time of taking each ultrasonic tomographic image,and data corresponding to each of the marks 82 are respectivelyextracted from the surface shape measurement data fetched in from thethree-dimensional shape measurement device 30, and based on theextracted data, the three-dimensional coordinates of the center of eachof the marks 36A at the time of taking each ultrasonic to tomographicimage, and the three-dimensional coordinates of the center of each ofthe intraoperative marks 82 are respectively obtained by calculation.Further, based on the three-dimensional coordinates of the center ofeach of the marks 36A at the time of taking each ultrasonic tomographicimage, the three-dimensional coordinates (coordinate values in the boxtype coordinate system) at the leading end of the probe 36 at the timeof taking each ultrasonic tomographic image, and the direction of theprobe 36 (the direction thereof in the box type coordinate system) arecalculated,

At step 138, based on data of the plurality of ultrasonic tomographicimages fetched in from the ultrasonic tomographic device 34, featurepoints (including points corresponding to feature portions of a brain,such as brain grooves, arteries, veins and so on, and also includingpoints corresponding the boundaries between a brain tumor and a healthyportion), which feature points are located within the brain (anunexposed portion in which the three-dimensional coordinates can bedetected by the three-dimensional shape measurement device 30) and arereadily recognized on an image, are respectively extracted from each ofultrasonic tomographic images. At step 140, first, based on thethree-dimensional coordinates at the leading end of the probe 36 at thetime of taking each ultrasonic tomographic image, and the direction ofthe probe 36, both of which are calculated in step 136, and also basedon the positions of the feature points on each of the ultrasonictomographic image, the three-dimensional coordinate of each featurepoint in the box type coordinate system are calculated. Subsequently,the coordinate conversion expression derived at the previous step 114 isread out from the HDD 20, and using the coordinate conversion expressionread therein, the three-dimensional coordinates of each feature point inthe box type coordinate system are converted to the three-dimensionalcoordinate values in the MRI coordinate system with the variousintraoperative marks 82 represented by the three-dimensional coordinatesobtained at step 136, and the three-dimensional coordinates of eachfeature point after having been subjected to the coordinate conversion,and the position of each feature point on the ultrasonic tomographicimage are stored, as unexposed area data, in the HDD 20. As a result,alignment of the second position information (unexposed area data), andthe three-dimensional brain model (and the MRI image) is completed.

When, due to the aforementioned processing, acquisition of surfacemeasurement data used for correction of an MRI image, and unexposed areadata has been completed, at the next step 142, data of athree-dimensional brain model (also refer to FIG. 6A) is fetched in fromthe HDD 20. At the subsequent step 144, matching between a producedimage and an MRI image (a determination that each of points of the brainsurface represented on the produced image corresponds to which portionon the MRI image) is carried out by collating the feature portion of thebrain appearing on the produced image represented by image data fetchedin from the video camera 32 (for example, brain grooves, gyrus of abrain, arteries, veins and so on) with the feature portions of the brainappearing on the MRI image. Further, in the present embodiment, theposition and direction of the video camera 32 are adjusted so as toallow image pickup of the same range as the measurement range of thethree-dimensional shape measurement device 30. Therefore, based on theresult of matching between the produced image and the MRI image, it isdetermined that each of the points of the brain surface in which thethree-dimensional coordinates in the MRI coordinate system are wellknown by the surface measurement data correspond to which portion on theMRI image. And then, by determining, based on the positions of the nodepoints (feature points) of the three-dimensional brain model stored inthe HDD 20 at the time of generating the three-dimensional brain model,a node point corresponding to each point on the surface of the brain inwhich the three-dimensional coordinates in the MRI coordinate system arewell known by the surface measurement data; matching between the surfacemeasurement data and the three-dimensional brain model is carried out.

As described above, due to the image obtained by the video camera 32being used for the matching between the surface measurement data and thethree-dimensional brain model, the matching between the surfacemeasurement data and the three-dimensional brain model can be carriedout by using, for example; features which are not clear on the surfacemeasurement data such as change of color in the surface of a brain.Therefore, the precision of matching between the surface measurementdata and the three-dimensional brain model can be improved.

Further, at step 144, due to the feature portions of the brainrepresented by an ultrasonic tomographic image being collated with thefeature portions of the brain represented by the MRI image in a mannersimilar to the above, it is determined that points corresponding to theinside of the brain in the ultrasound tomographic image correspond towhich portion on the MRI image, and based on the positions of the nodepoints (feature points) of the three-dimensional brain model on the MRIimage, and the positions of feature points extracted from the ultrasonictomographic image on the ultrasonic tomographic image, a node pointcorresponding to each of feature points within the brain in which thethree-dimensional coordinates in the MRI coordinate system are wellknown by the unexposed area data from among various node pointsconstituting the three-dimensional brain model.

Due to the three-dimensional coordinates of the node point which isdetermined as that corresponds to any of the points on the brain surfacerepresented by the surface measurement data being replaced bythree-dimensional coordinates of the corresponding point (thethree-dimensional coordinates in the MRI coordinate system representedby the surface measurement data), and also due to the three-dimensionalcoordinates of the node point which is determined as that corresponds toany of the feature points within the brain represented by the unexposedarea data being replaced by the three-dimensional coordinates of thecorresponding feature point (the three-dimensional coordinates in theMRI coordinate system represented by the unexposed area data), as isalso shown in FIG. 6B for instance, the position of the node pointcorresponding to any one of the points on the brain surface representedby the surface measurement data or any one of the feature points withinthe brain represented by the unexposed area data is corrected, fromamong the node points constituting the three-dimensional brain model.Incidentally, FIG. 6B shows an example in which the position correctionis carried out only for node points corresponding to the front surfaceor rear surface of the brain, but node points corresponding to a regionbetween the front and rear surfaces of the brain may also be intendedfor the correction of the positions thereof.

At step 146, due to, based on the node points intended for correction ofthe positions thereof and the corrected positions of the node points atstep 144, external force that causes the node points targeted forcorrection of the positions thereof at step 144, from among the nodepoints constituting the three-dimensional brain model, to move to thecorrected positions at step 144 being applied to the three-dimensionalbrain model, the way in which the positions of other node points aredisplaced is estimated and calculated by applying a finite elementmethod, and based on the result of the estimation and calculation, asalso shown in FIG. 6C for instance, the positions of node points otherthan the node points targeted for correction of the positions thereof atstep 144 are corrected. As a result, the three-dimensional brain modelcan be corrected so as to represent the current state of the brain(displacement or distortion of each of the parts thereof) with a highdegree of precision. Incidentally, a method similar to the finiteelement method (for example, a simplified method of the finite elementmethod, which is intended for high speed processing and the like) mayalso be applied in place of the finite element method.

At the subsequent step 148, based on the three-dimensional brain modelin which the positions of the node points are corrected at steps 144 and146, and the positions of the node points (feature points) of thethree-dimensional brain model on the MRI image, Geometrical conversionis carried out for the MRI image so as to, for examples allow thepositions of pixels of the MRI image to move in accordance with themovement of the positions of the node points based on the correction ofthe three-dimensional brain model, whereby the MRI image is correcteddepending on the displacement or distortion of various parts of thebrain represented by the corrected three-dimensional brain model. As aresult, a high-definition MRI image that represents the current state ofthe brain with a high degree of precision can be obtained.

As a result, by referring to the aforementioned MRI images updated andshown on the display 18, the surgeon can make a correct diagnosis of theposition of a brain tumor to be removed, and the like even if the partsof a brain are displaced or distorted due to various surgical operationsafter cranial surgery. Further, the aforementioned correction andupdate/display of MRI images are carried out repeatedly (each time thedecision of step 118 is made affirmative) until the surgical operationis finished (until the decision of step 120 is made affirmative),Therefore, the surgeon can perform the surgical operation, by referringto the MRI images which are updated and displayed as needed, whileconfirming a proper anatomical positional relationship between theregion for which the surgical operation has been performed, and aperipheral brain region at various stages, for example, at the time whenremoval of the brain tumor is started, or at the time when the removaloperation is completed. Further, the presence of a remaining- tumor(rest of the tumor) can be confirmed, thereby making it possible tocompletely remove the brain tumor to be removed. Moreover, the MRI imageshown on the display 18 may be a functional mapping MRI image in whichthe distribution state of various functional fields as searched inadvance is indicated as a map in a superimposed manner. However, whenthe functional mapping MRI image is displayed, the surgeon can proceedwith the surgical operation while keeping track of the positionrelationship between the region for which the surgical operation isperformed, and each of the functional fields.

INDUSTRIAL APPLICABILITY

In this manner, in the surgical operation supporting apparatus 10according to the present embodiment, based on the surface configurationdata obtained by optically measuring the surface of a brain using thethree-dimensional shape measurement device 30 during surgery (and bycarrying out image pickup of the surface of the brain using the videocamera 32 during surgery), and unexposed area data obtained by measuringan unexposed area of the brain with ultrasonic waves by means of theultrasonic tomographic imaging device 34 during surgery, an MRI image aspreviously picked up is corrected and the correct MRI image thatrepresents the current state of the brain with a high degree ofprecision is displayed during the surgical operation, Accordingly, it ispossible for the surgeon to recognize the current state of the brain(the state of the brain after displacement or distortion occurs duringthe surgical operation and so on), and realize improvement in theprecision of the surgical operation.

Further, in the surgical operation supporting apparatus 10 according tothe present embodiment, the operation of measuring the surfaceconfiguration using the three-dimensional shape measurement device 30and the image pickup operation of the video camera 32 are completed over20 seconds or thereabouts and the image pickup operation of a pluralityof ultrasonic tomographic images using the ultrasonic tomographic device34 is completed over 3 seconds or thereabouts, Therefore, compared withthe case in which MRI images are periodically taken during the surgicaloperation, the downtime of the surgical operation is significantlyreduced. Accordingly, it is possible to avoid the situation in which thesurgical operation is interrupted due to MRI images which represent theactual state of the brain being displayed during the surgical operation.

Moreover, the surgical operation supporting apparatus 10 according tothe present invention can be realized only by adding thethree-dimensional shape measurement device 30, the video camera 32, andthe computer 12 having a three-dimensional model generation pro-ram andan MRI image display program installed thereon. Therefore, compared withthe case in which MRI images are periodically taken during surgery, anapparatus of far less cost can be realized.

In the foregoing, the structure in which only one video camera 32 isprovided was described above, but the present invention is not limitedto the same. A plurality of video cameras used for carrying out imagepickup from different directions may be provided, and due to imagesproduced by these video cameras being used to carry out matching betweensurface measurement data and a three-dimensional brain model, theprecision of matching between the surface measurement data and thethree-dimensional brain model may be further improved.

Further, there was shown above a case in which the present invention isapplied to supporting of a surgical operation for removing a braintumor, but the present invention is not limited to the same and may alsobe applied to any surgical operation of the brain other than that forremoving the brain tumor. Further, the operation site is not limited tothe brain region, and the present invention is also applicable tosupporting of a surgical operation for any body part.

Moreover, there was described above a case in which an MRI image is usedas a high-definition tomographic image according to the presentinvention. However, any tomographic image that represents an operationsite with a high degree of definition may be applied, for example,tomographic images produced by using other known image pickup methodsuch as X-ray CT imaging or the like. Further, when a surgical operationis performed while referring to other tomographic images produced byother image pickup methods (for example, positron emission tomography(PET) or single photon emission computer tomography (SPECT)) in additionto the high-definition tomographic image according to the presentinvention, the other tomographic images are previously made tocorrespond to the high-definition tomographic images according to thepresent invention, and after the high-definition tomographic imagesaccording to the present invention are corrected based on the surfacemeasurement data and unexposed area data as described above, theaforementioned other tomographic images may also be corrected anddisplayed based on the corrected high-definition tomographic images.

EXPLANATION OF REFERENCE NUMERALS

-   10 surgical operation supporting apparatus-   12 computer-   18 display-   22 drive-   24 MRI imaging device-   26 surgical microscope-   30 three-dimensional shape measurement device-   32 video camera-   34 ultrasonic tomographic device-   36 probe-   80 preoperative mark-   82 intraoperative mark

1. A surgical operation supporting apparatus comprising: firstacquisition means that optically measures a surface of an operation siteduring surgery and that acquires first position information representinga three-dimensional position of each of points on the surface of theoperation site; second acquisition means that measures an unexposedportion of the operation site with ultrasonic waves during surgery andthat acquires second position information representing athree-dimensional position of each of points in the unexposed portion ofthe operation site; correction means that, based on the first positioninformation acquired by said first acquisition means and the secondposition information acquired by said second acquisition means,estimates displacement and distortion at each of the points in theoperation site using a three-dimensional model generated based on aplurality of high-definition tomographic images of the operation site,which images are taken before surgery, and that corrects the pluralityof high-definition tomographic images; and display control means thatallows the high-definition tomographic images corrected by saidcorrection means to be shown on display means.
 2. The surgical operationsupporting apparatus according to claim 2, wherein said firstacquisition means comprises a scanning device mounted at a surgicalmicroscope and scanning the surface of the operation site with laserlight, and detecting means mounted at the surgical microscope andreceiving laser light reflected by the surface of the operation site,thereby detecting a three-dimensional position of a portion on which thelaser light is irradiated, on the surface of the operation site, and anoperation of detecting the three-dimensional position by said detectingmeans is carried out repeatedly while scanning each of the points on thesurface of the operation site with laser light, thereby acquiring thefirst position information.
 3. The surgical operation supportingapparatus according to claim 1, wherein said first acquisition meansfurther comprises image pickup means mounted at the surgical microscopeand producing images of the surface of the operation site, and saidcorrection means is provided so as to estimate displacement anddistortion at each of the points in the operation site also using imagesproduced by said image pickup means.
 4. The surgical operationsupporting apparatus according to claim 1, wherein said secondacquisition means comprises a probe that transmits ultrasonic waves tothe operation site and receives ultrasonic waves reflected by the pointsin the unexposed portion of the operation site, and conversion meansthat converts the ultrasonic waves received by the probe to tomographicimages, and said second acquisition means is provided so as to acquirethe second position information by obtaining the three-dimensionalposition of each of the points on the ultrasonic tomographic imagesobtained by said conversion means.
 5. The surgical operation supportingapparatus according to claim 4, wherein: said first acquisition meanscomprises a scanning device mounted at a surgical microscope andscanning the surface of the operation site with laser light anddetecting means mounted at the surgical microscope and receiving laserlight reflected by the surface of the operation site, thereby detectinga three-dimensional position of a portion on which the laser light isirradiated, on the surface of the operation site, and said firstacquisition means also detects the three-dimensional position of theprobe of said second acquisition means; and said second acquisitionmeans obtains, based on the three-dimensional position of the probedetected by said first acquisition means, the three-dimensional positionof each of the points on the ultrasonic tomographic image.
 6. Thesurgical operation supporting apparatus according to claim 1, whereinthe high-definition tomographic image is an MRI image produced bynuclear magnetic resonance-computed tomography.
 7. The surgicaloperation supporting apparatus according to claim 1, wherein saidcorrection means corrects, based on the first position informationacquired by said first acquisition means and the second positioninformation acquired by said acquisition means, a position of a portionwhose three-dimensional position is known by the first positioninformation and the second position information in the three-dimensionalmodel of the operation site, and thereafter, estimates displacement anddistortion at a portion whose three-dimensional position is not known inthe three-dimensional model of the operation site, by means of a finiteelement method or a method similar thereto, and based on the estimatedresult, recorrects the three-dimensional model of the operation site,and further based on the recorrected three-dimensional model of theoperation site, carries out correction of the plurality ofhigh-definition tomographic images.
 8. The surgical operation supportingapparatus according to claim 1, wherein when the plurality ofhigh-definition tomographic images are produced before a surgicaloperation, at least three first marks are applied on the periphery ofthe operation site, and at the time of the surgical operation, at leastthree second marks are applied to the vicinities of the operation site;said first acquisition means further acquires mark position informationthat represents respective three-dimensional positions of the firstmarks and the second marks; said correction means carries out, based onthe mark position information acquired by said first acquisition means,and positions of image portions corresponding to the first marks on thehigh-definition tomographic image, alignment of the high-definitiontomographic image and the first position information and the secondposition information.
 9. The surgical operation supporting apparatusaccording to claim 1, wherein operation of acquiring the first positioninformation by said first acquisition means, acquiring the secondposition information by said second acquisition means, correcting theplurality of high-definition tomographic images by said correctionmeans, and displaying the high-definition tomographic images by saiddisplay means is carried out repeatedly during the surgical operation.10. A surgical operation supporting method comprising: a first step inwhich based on a plurality of high-definition tomographic images of anoperation site taken as an image before surgery: a three-dimensionalmodel of the operation site is generated; a second step in which asurface of the operation site is optically measured during surgery, soas to acquire first position information that represents athree-dimensional position of each of points on the surface of theoperation site, and an unexposed portion of the operation site ismeasured with ultrasonic waves during surgery, so as to acquire secondposition information that represents a three-dimensional position ofeach of points of the unexposed portion in the operation site; a thirdstep in which based on the first position information and the secondposition information acquired by said second step, displacement anddistortion at each of the points in the operation site are estimatedusing the three-dimensional model generated by said first step, and inaccordance with the estimated displacement and distortion at each of thepoints in the operation site, the plurality of high-definitiontomographic images of the operation site taken as images before surgeryare corrected; and a fourth step in which the high-definitiontomographic images corrected by said third step are shown on displaymeans.
 11. A surgical operation supporting program that causes acomputer, to which display means is connected, to function as: firstacquisition means that optically measures a surface of an operation siteduring surgery and that acquires first position information representinga three-dimensional position of each of points on the surface of theoperation site; second acquisition means that measures an exposedportion of the operation site with ultrasonic waves during the surgeryand that acquires second position information representing athree-dimensional position at each of points in the unexposed portion ofthe operation site, correction means that, based on the first positioninformation acquired by said first acquisition means and the secondposition information acquired by said second acquisition means,estimates displacement and distortion at each of the points in theoperation site using a three-dimensional model generated based on aplurality of high-definition tomographic images obtained before thesurgery, and in accordance with the estimated displacement anddistortion occurring at each of the points in the operation site,corrects the plurality of high-definition tomographic images of theoperation site, which images are produced before the surgery; anddisplay control means that causes the high-definition tomographic imagescorrected by said correction means to be shown on display means.